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LONG-TERM  GOALS 


We  put  forth  the  idea  of  developing  a  statistical  model  representing  the  relationship 
between  model  parameters  and  macroscopic  features  in  meteorological  quantities,  with  the 
latter  quantified  in  terms  of  measures  commonly  used  in  spatial  verification  methods.  For 
example,  if/when  a  spatial  verification  method  suggests  that  frontal  speed  was  forecast  in¬ 
correctly,  we  would  like  to  be  able  to  set  the  model  parameters  (e.g.,  diffusion  rate)  to 
remedy  that  problem  in  some  optimal  sense.  This  primary  goal  requires  the  identification  of 
relevant  model  parameters,  which  in  turn  requires  performing  sensitivity  analysis  (our  sec¬ 
ondary  goal).  To  validate  the  results  of  the  variance-based  sensitivity  analysis,  comparisons 
are  made  with  a  more  traditional  method  based  on  adjoints.  After  the  “important”  model 
parameters  are  identified,  an  emulator  will  be  developed.  An  emulator  consists  of  a  statis¬ 
tical  model  which  represents  the  relationship  between  model  parameters  and  macroscopic 
features  (e.g.,  spatial  structure)  of  forecast  fields.  Such  an  emulator  (and  its  inverse)  has 
both  scientific  value  and  practical  utility.  An  example  of  the  former  is  the  knowledge  gained 
from  identifying  the  statistical  relations  between  model  parameters  and  forecast  parameters, 
and  model-tuning  is  an  example  of  the  practical  utility. 
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OBJECTIVES 


1.  Perform  variance-based  and  adjoint  sensitivity  analysis  of  the  Lorenz  1963  model, 
in  preparation  for  application  to  COAMPS.  The  sensitivities  are  computed  with  respect  to 
both  initial  conditions  and  model  parameters,  but  only  the  latter  are  used  as  inputs  to  the 
emulator. 

2.  Develop  emulators  to  represent  the  complex,  nonlinear  relationship  between  model 
parameters  and  forecast  features  arising  in  spatial  verification  techniques. 


APPROACH 


The  specific  type  of  sensitivity  analysis  we  have  attempted  to  tailor  to  our  needs  is 
called  variance-based  sensitivity  analysis  (Saltelli  2010).  We  have  applied  the  method  to 
both  COAMPS  and  the  Lorenz  1963  model.  Applying  the  method  to  a  complex  and  a 
simple  model  at  the  same  time  has  been  fruitful  in  explaining  some  of  our  COAMPS  results. 

The  emulators  we  have  developed  (trained)  thus  far  are  still  tentative,  because  the  sen¬ 
sitivity  analysis  work  is  still  in  progress.  We  have  tested  polynomial  regression  and  neural 
networks  as  emulation  models,  and  they  both  seem  to  be  adequate  for  the  task  at  hand. 

The  variance-based  sensitivity  analysis  approach  and  the  emulation  method  are  closely 
tied  (Oakley  and  O’Hagan  2004,  Rouigier  2005).  This  connection  between  variance-based 
sensitivity  analysis  and  model  emulation  is  the  main  reason  behind  our  approach,  because 
it  is  the  emulator  which  is  ultimately  used  for  model  tuning. 


WORK  COMPLETED 


Various  objectives  are  being  addressed  in  parallel.  Both  variance-based  and  adjoint 
sensitivity  analysis  of  Lorenz  1963  has  been  done,  but  the  comparison  between  them  is 
not  complete.  The  variance-based  results  for  the  Lorenz  1963  model  and  COAMPS  were 
presented  at  a  seminar  at  NRL  on  October  10,  2011.  Emulators  have  been  trained  for  both 
Lorenz  1963  and  COAMPS,  but  they  are  all  in  developmental  stage. 

We  have  initiated  a  collaboration  with  an  expert  in  adjoint  sensitivity  analysis  (Greg 
Hakim,  University  of  Washington).  We  have  discovered  that  in  fact  the  variance-based  and 
the  adjoint  method  may  be  quite  different  in  terms  of  their  assumptions  and  goals,  almost 
to  the  point  of  making  a  direct  comparison  very  difficult  (or  nearly  impossible).  Through 
the  collaboration  with  Greg  Hakim,  we  hope  to  address  this  question  of  how  to  perform  the 
comparisons.  Thus  far  that  work  has  focused  on  the  Lorenz  model.  Although  the  Lorenz 
model  is  qualitatively  different  from  COAMPS  (because  the  Lorenz  adjoint  exists  in  ana¬ 
lytic  form,  but  only  a  numerical  solutions  exists  for  the  COAMPS  adjoint),  the  comparison 
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of  the  two  sensitivity  analysis  methods  (variance-based  vs.  adjoint)  on  the  Lorenz  model  is 
important  for  understanding  the  COAMPS  results. 


RESULTS 


The  Lorenz  1963  model  is  well- studied:  State  variables  X,  Y,  Z  are  coupled  to  one  another 
in  a  first-order  (in  time)  differential  equation,  involving  parameters  s,r,b.  To  perform  sensi¬ 
tivity  analysis,  one  considers  a  reference  trajectory  denoted  Xr,  Yr,  Zr ,  with  respect  to  which 
perturbations  SX,  SY,  SZ  are  introduced.  The  perturbations  may  be  caused  by  uncertainty 
in  the  initial  conditions  (<5X0,  SY0,  SZ0 )  or  in  the  model  parameters  (8s,  Sr,  8b).  One  question 
is  to  assess  the  manner  in  which  the  perturbations  SX,  8Y,  5Z  depend  on  8Xq,  8Y0, 8Z0,  and 
5s,  8r,  8b.  Note  that  we  are  considering  not  only  sensitivity  to  initial  conditions,  but  also 
sensitivity  to  the  perturbations  (uncertainty)  in  the  model  parameters. 

We  used  a  method  developed  by  one  of  the  pioneers  of  the  adjoint  method  (Hall  1986); 
this  formulation  of  the  adjoint  method  has  the  virtue  of  allowing  one  to  derive  simple  math¬ 
ematical  expressions  which  can  aid  in  understanding  the  complexity  brought  about  by  the 
nonlinear  nature  of  the  Lorenz  model.  Specifically,  we  have  derived  the  equations  for  the 
time-average  of  the  perturbations.  For  example,  the  time-average  of  SZ,  over  a  period  from 
0  to  r  is 


8Z  =  /;  [Vi(Yr  -  Xr)  8s  +  v2XR  Sr  +  v3ZRSb  ]  dt  (1) 

-  vi (t  =  0)  SXQ  -  v2(t  =  0)  8Y0  -  v3(t  =  0)  SZ0 

The  Vj(t)  are  solutions  to 


where  N*  is  the  matrix  defining  the  adjoint  of  the  tangent  linear  model;  it  is  well-known  and 
can  be  written  in  analytic  form  (not  shown  here).  The  adjoint  solutions  vt  must  satisfy  the 
condition  Vi(t  =  r)  =  0. 

Solving  the  adjoint  equation  (2)  “backward”  in  time  gives  v^ft  =  0),  which  then,  ac¬ 
cording  the  equation  (1)  are  precisely  the  sensitivities  with  respect  to  initial  conditions. 
Equation  1  also  gives  the  sensitivities  with  respect  to  model  parameters,  namely  the  coeffi¬ 
cients  of  8s,  Sr,  8b.  These  too  involve  the  adjoint  solutions,  but  integrated  over  time. 

The  simple  and  explicit  nature  of  this  formulation  is  precisely  what  is  required  for  com¬ 
paring  it  with  the  variance-based  method.  We  are  currently  casting  the  variance-based 
method  in  a  form  that  allows  comparison  with  Eq.  1.  One  difference  between  the  two  meth¬ 
ods  (which  makes  comparisons  difficult)  is  that  the  variance-based  method  explicitly  contain 
interactions  between  the  various  perturbations,  whereas  the  adjoint  method  (1)  does  not. 

Turning  to  numerical/simulations  (important  for  transitioning  to  COAMPS),  Fig.  1 
shows  the  sensitivities  defined  above  as  a  function  of  time.  Apart  from  the  variations  in 
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time,  the  relative  vertical  displacement  between  the  three  curves  in  the  left  panel  clearly 
shows  that  the  three  parameters  have  very  different  effects.  By  contrast,  the  sensitivities 
with  respect  to  initial  conditions  (right  panel)  are  initially  different,  but  quickly  approach  a 
common  value  of  zero. 


Figure  1.  Left:  Sensitivity  of  5Z  to  model  parameters  5s  (black),  5r  (red),  and  5b  (green). 
Right:  sensitivity  of  5Z  to  initial  conditions  5X0  (black),  5Y0  (red),  and  5Z0  (green).  The 
x-axis  in  both  plots  is  time. 

Fig.  2  shows  the  same  sensitivities,  but  as  a  function  of  the  X  and  Z  state  variables  of 
the  model.  One  task  at  hand  is  to  compare  these  results  with  those  obtained  via  the  adjoint 
code  for  the  Lorenz  63  model.  An  example  is  shown  in  Fig.  3  (already  produced  by  Greg 
Hakim  as  part  of  a  class  taught  at  the  University  of  Washington).  The  sensitivity  metrics 
used  in  the  two  analyses  (the  analytic  method  above,  and  that  based  on  the  adjoint  code)  are 
different  because  of  the  way  the  two  methods  compute  sensitivity.  For  example,  the  metric 
in  the  latter  is  required  to  be  a  positive  definite  quantity,  but  no  such  restriction  is  required 
in  the  variance-based  method.)  We  are  currently  working  to  use  similar  metrics  in  all  the 
methods  being  compared. 


Figure  2.  Same  sensitivities  as  in  Fig,  1,  but  as  a  function  of  the  X  and  Z  state  variables. 
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Figure  3.  Sensitivity  according  to  the  adjoint  code  for  the  Lorenz  1963  model.  Produced 
here  with  permission  of  Greg  Hakim. 


IMPACT  /APPLICATIONS 


The  impact  of  our  work  on  the  Lorenz  1963  model  is  likely  to  be  mostly  of  academic 
and  scientific  value.  Its  main  purpose  here  is  to  guide  the  work  on  COAMPS.  By  contrast, 
the  practical  values  of  the  sensitivity  analysis  of  COAMPS  and  the  emulator  are  likely  to 
be  significant.  Together  they  allow  one  to  assess  the  effect  of  model  parameters  on  spatial 
forecast  verification,  and  vice  versa.  As  such,  not  only  the  effects  of  changing  the  parameters 
can  be  anticipated,  but  also  desirable  properties  of  forecasts  can  be  assured  (in  some  optimal 
sense)  by  tuning  the  parameters  accordingly. 
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